Carnival-combining speech technology and computer animation.

Speech is powerful information technology and the basis of human interaction. By emitting streams of buzzing, popping, and hissing noises from our mouths, we transmit thoughts, intentions, and knowledge of the world from one mind to another. We’re accustomed to thinking of speech as an acoustic, auditory phenomenon. However, speech is also visible. Although the primary function of speech is to manipulate air in the vocal tract to produce sound, this action has an ancillary effect of changing the face’s appearance. In particular, the action of the lips and jaw during speech causes constant deformation of the facial surface, generating a robust visual signal highly correlated with the acoustic one—that is, visual speech. In computer animation, this means that speech is something that must be studied, understood, and simulated. Speech animation, or lip synchronization, is a major challenge to animators, owing to its intrinsic complexity and viewers’ innate sensitivity to the face. (Actually, the term lip synchronization—lip sync—incorrectly implies that only the lips move, whereas actually almost all the facial surface below the eyes gets deformed during speech.) At the same time, demand for lip sync is sharply increasing, in terms of both realism and quantity. Automated solutions are now absolutely necessary. (For more on why this is the case, see the sidebar.) The past two decades have seen the emergence of techniques for animating speech automatically using speech technology—an interdisciplinary concept called visual speech synthesis. Since the late 1980s, two applications have been in development. Audio-driven animation automatically synthesizes facial animation from audio. Text-driven animation (or audiovisual text-to-speech synthesis) synthesizes both auditory and visual speech from text. The former is used for automatic lip sync with recorded audio, the latter for entirely text-based avatars. But speech technology and computer graphics remain worlds apart, and the development of visual speech synthesis suffers from lack of a unifi ed conceptual and technological framework. To meet this need, researchers at Speech Graphics (www. speech-graphics.com) and the University of Edinburgh’s Centre for Speech Technology Research (CSTR; www.cstr.ed.ac.uk) are developing Carnival, an object-oriented environment for integrating speech processing with real-time graphics. Carnival comprises an unlimited number of modules that can be dynamically loaded and assembled into a mutable animation production system.